System for ray tracing augmented objects

ABSTRACT

The present disclosure describes a new global illumination ray tracing, concentrated at augmented objects of virtual or augmented reality, utilizing the graphics pipeline. Secondary rays are handled in large groups, originating at clusters of primary hit points, and intersecting with scene geometry.

FIELD OF THE INVENTION

The present invention relates to new and improved ways of carrying outray tracing method at a reduced computational complexity.

BACKGROUND OF THE INVENTION

Augmented reality (AR) is a live view of a physical, real-worldenvironment whose elements are augmented by computer-generated graphics.Information about the augmented objects is overlaid onto the real world.Augmentation is conventionally in real time and in visual context withenvironmental elements. With the help of advanced AR technology, theinformation about the surrounding real world of the user becomesinteractive and digitally manipulable. Because AR brings out thecomponents of the digital world into a person's perceived real world,the user experiences and interacts with the augmented reality, dependingon the level of photo-realism of the augmented objects and theirintegration with the real environment.

In prior art, the graphics used in AR were based on the conventionalraster technology. However, raster graphics are very mediocre due to thetradeoff of quality and speed, in favor of speed. Its visual contextlacks such necessary elements of photo-realism as reflections,refractions, color bleeding, color bleeding, caustics, etc. In priorart, a high quality computer-generated realism is found in the filmindustry, enabled through the use of ray tracing computer-generatedgraphics.

Ray tracing is a computer graphics technique for generating images bytracing the rays of light and simulating the effects of their encounterswith virtual objects. The idea behind ray-tracing is to findmathematical solutions to compute the intersection of a ray with varioustypes of geometry, by solving the visibility between points. Thistechnique is capable of producing a high visual realism. In low-end raytracing, light sources are the only illuminating objects in the scene.

More realistic imaging is delivered by a high-end ray tracing, calledpath tracing, which is based on global illumination. Global illuminationtakes into account not only the light which comes directly from a lightsource, but also subsequent cases in which light is returned by othersurfaces in the scene, whether reflective or not.

Path tracing, referred to as Monte Carlo ray tracing, renders a 3D sceneby randomly tracing samples of possible light paths. Repeated samplingfor any given pixel in the image will eventually cause the average ofthe samples to converge to the correct solution of a rendering equation,making it one of the most physically accurate 3D graphic renderingmethods in existence. Path tracing can generate images that are faithfulto reality, and are indistinguishable from photographs (e.g. The Avatarmovie). The visual quality is higher than that of ray tracing, but at amuch greater computational cost.

The most time-consuming tasks in the ray tracing of prior art aretraversals of acceleration structures, as well as intersection testsbetween rays and polygons. Every single ray is traversed across anaccelerating structure (e.g. K-trees or BVH trees), seeking polygonsthat are candidates for intersection. These traversals become a majortime-consuming action—they typically take 60%-70% of the imagegeneration time. Then, all candidate polygons associated with the searchmust undergo a line-triangle intersection test, to determine theearliest hit along the ray's path.

The layout of the prior art ray tracing method is depicted in FIG. 1.First, an acceleration structure must be constructed 10. Theconstruction is done as a preprocessing step, and takes much more timethan generating a single image. Generally, the construction time dependson the scene size. The bigger the scene, the longer the constructiontime. Every major modification in the scene necessitates areconstruction of the acceleration structure. The memory size istypically doubled by the acceleration structure. Tracing of rays 12 isbased on massive traversals of the acceleration structure 11, when eachray is traversed across the structure in search of intersections betweenthe ray and various scene objects. The resulting intersection points arelighted, textured, shaded 13, and aggregated into image pixels.

There are two major drawbacks associated with the use of accelerationstructures in ray tracing of prior art; (i) they must be repeatedlyreconstructed for scene changes, and (ii) traversals of these structuresare time-consuming. Both disadvantages contradict with the real-timerequirements of AR.

Therefore, the primary object of the present invention is to acceleratethe performance of global illumination ray tracing up to real time,making it suitable for AR.

Another object of the present invention is to reduce the computationalcomplexity of ray tracing.

Another object of the present invention is to reduce the powerconsumption of ray tracing.

Another object of the present invention is to enable global illuminationray tracing by the processing level of consumer computing devices.

SUMMARY OF THE PRESENT INVENTION

Some embodiments of the current invention are applied to both augmentedreality (AR) and virtual reality (VR). AR is a live view of a physical,real-world environment whose elements are augmented bycomputer-generated graphics. VR replaces the real world with a simulatedone. Augmentation is conventionally in real time and in a visual contextwith environmental elements. The user's experience and interaction aredirectly influenced by the level of realism in the AR and VR.

In the prior art, the imaging of augmented objects is produced by theconventional raster graphics, due to its high speed. With the help ofadvanced AR technology (e.g. adding computer vision and objectrecognition), the information about the surrounding real world of theuser becomes interactive and digitally manipulable. Information aboutthe environment and its objects is overlaid onto the real world.

However, the ordinary raster graphics technology is mediocre in imagequality and in visual context with the real-world environment. Thedesired high quality of computer-generated realism can be found today inthe film industry, and is enabled through the use of the globalillumination ray tracing, namely path tracing. Unfortunately, it doesnot fit the AR because of a very high computational complexity, whichcauses long production times and requires expensive computing farms.

The present invention teaches an innovative way of delivering pathtracing in real time, having reduced computational complexity and powerconsumption, and suitable to the processing level of consumer computingdevices. Aspects of the present invention enable to concentrate onselected object(s), generating a photo-realistic image of the objectrealistically overlaid with a preset environment.

In the embodiments of the present invention, the Acceleration Structuresof prior art are replaced by a new and novel device—a DynamicallyAligned Structure (DAS). DAS is a means for carrying out theintersection between the secondary rays and the scene geometry in largegroups of rays, gaining a high speed and a reduced computationalcomplexity.

FIG. 2 shows the stages of the present invention's path tracing. Themain difference as compared to the prior art (FIG. 1) is the lack ofacceleration structures. These structures are replaced by the DAS device21. There is no pre-processing for reconstruction, and no traversals ofacceleration structures.

The DAS is an aligned projection of rays, used to carry secondary raysassociated with existing hit points. Instead of an individual shootingof secondary rays for every single hit point (or in small packets ofrays), as in the prior art, we do it collectively, cutting the cost.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of non-limiting examples only,with reference to the accompanying figures and drawings, wherein likedesignations denote like elements. Understanding that these drawingsonly provide information concerning typical embodiments of the inventionand are not therefore to be considered limiting in scope:

FIG. 1. Prior art. Block diagram of ray tracing.

FIG. 2. Block diagram of ray tracing of the present invention.

FIG. 3. Prior art. Secondary rays generating global illumination at apoint of ray/object intersection.

FIG. 4. The basic mechanism of DAS.

FIG. 5. Multiple DAS projections shot in randomly varying directions.

FIG. 6. Generating hit points by successive DAS projections.

FIG. 7a . Opening scene consisting of two triangles and two primaryHIPs.

FIG. 7b . An early segment of a DAS ray shot towards a HIP.

FIG. 7c . A main segment of a DAS ray carries a secondary ray, shot fromthe HIP.

FIG. 8a . DAS renders HIPs data only.

FIG. 8b . DAS renders the scene geometry. Objects along the earlysegment of DAS ray are discarded.

FIG. 8c . DAS flowchart.

FIG. 9. Various cases of secondary rays carried by a single DASprojection

FIG. 10. An augmented object standing on a real desk of asemi-reflective surface.

FIG. 11. Direct imaging of augmented object. Primary rays are shot fromthe camera and tilted at varying directions.

FIG. 12a . Direct imaging of augmented object. Secondary rays aregenerated by multiple DAS shots of a perspective projection.

FIG. 12b . Direct imaging of augmented object. Secondary rays aregenerated by multiple DAS shots of parallel projection, saving renderingof a bigger data.

FIG. 12c . Flowchart of generating direct image of an augmented object.

FIG. 13a . Reflected imaging of augmented object. Primary rays are shotat the reflection area, repeated and tilted multiple times.

FIG. 13b . Reflected imaging of the augmented object. The secondaryrays, carried by DAS projection, are shot at the object through thecluster of primary HIPs.

FIG. 14a . Reflected imaging of the augmented object. The multiple DASprojections are randomly tilted.

FIG. 14b . Reflected imaging of the augmented object. The contributionof a secondary ray to the aggregated light energy comply with the BRDFfunction.

FIG. 14c . Flowchart of reflected imaging.

FIG. 15a . Color bleeding effect of the augmented object on itsenvironment. Primary rays.

FIG. 15b . Color bleeding effect of the augmented object on itsenvironment. Secondary rays.

FIG. 15c . Flowchart of generating color bleeding effect.

FIG. 16. Collecting the sampled light values at the pixel of origin.

FIG. 17. Hardware for AR and VR.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The principles and operation of an apparatus according to the presentinvention may be understood with reference to the figures and theaccompanying description wherein similar components appearing indifferent figures are denoted by identical reference numerals. Thedrawings and descriptions are conceptual only. In actual practice, asingle component can implement one or more functions; alternatively,each function can be implemented by a plurality of components anddevices. It will be readily understood that the components of thepresent invention, as generally described and illustrated in the figuresherein, could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the apparatus, system, and method of the presentinvention, as represented in the figures herein, is not intended tolimit the scope of the invention, as claimed, but is merelyrepresentative of embodiments of the invention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions, utilizing terms such as “processing”, “computing”,“calculating”, “determining”, “generating”, “creating” or the like,refer to the action and/or processes of a computer or computing system,or processor or similar electronic computing device, that manipulateand/or transform data represented as physical, such as electronic,quantities within the computing system's registers and/or memories intoother data, similarly represented as physical quantities within thecomputing system's memories, registers or other such informationstorage, transmission or display devices.

Embodiments of the present invention may use terms such as processor,computer, apparatus, system, sub-system, module, unit, and device (insingle or plural form) for performing the operations herein. This may bespecially constructed for the desired purposes, or it may contain ageneral purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Several technical termsspecifically associated with our disclosure are herein defined.

Computer graphics pipeline refers to the most common form of computer 3Drendering, 3D polygon rendering, distinct from raytracing, andraycasting. In particular, in raycasting, a ray originates at the pointwhere the camera resides, if that ray hits a surface, then the color andlighting of the point on the surface where the ray hit is calculated. In3D polygon rendering the reverse happens, the area that is in view ofthe camera is calculated, and then rays are created from every part ofevery surface in view of the camera and traced back to the camera. Thegraphics pipeline is usually used in real-time rendering.

Rendering a projection is a 3D computer graphics process ofautomatically converting 3D wire frame models into 2D images renderingon a computer. The projection can be of a perspective, parallel, inverseor of another shape.

Render target is a feature of modern graphics processing units (GPUs)that allows a 3D scene to be rendered to an intermediate memory buffer,or Render Target Texture (RTT), instead of the frame buffer or backbuffer. This RTT can then be manipulated by pixel shaders in order tomake searches or apply effects to the final image.

Primary rays, are the first generation of rays in ray tracing, cast intothe scene from camera or from eye to solve for visibility, i.e. to findwhether the primary ray intersects a surface.

Secondary rays in ray tracing are spawned from primary rays at theyray-polygon intersection points. They are used to compute things likeshadows, reflections, refractions, etc. Herein, we use the termcollectively for all successive generations as well, such as for ternaryrays, forth generation, etc.

Global illumination, is a general name of a group of algorithms used in3D computer graphics that are meant to add more realistic lighting to 3Dscenes, taking into account not only the light that comes directly froma light source (direct illumination), but also subsequent cases in whichlight rays from the same source are reflected by other surfaces in thescene, whether reflective or not (indirect Ilumination).

Color bleeding in computer graphics is the phenomenon in which objectsor surfaces are colored by reflection of indirect light from nearbysurfaces. This is a visible effect that appears when a scene is renderedwith full global illumination.

Accelerating structures, such as grids, octrees, binary spacepartitioning trees (BSP trees), kd-trees and BVHs (bounding volumehierarchy), are used in ray tracing to solve for visibility, allowingimprovement of render times in speed and efficiency, as compared tonaïve ray tracing without accelerating structures.

GPGPU (general-purpose computing on graphics processing units) is theuse of a graphics processing unit (GPU), which typically handlescomputation only for computer graphics, to perform computation inapplications traditionally handled by the central processing unit (CPU).

Preset scene in AR, replaces the real time world. It is a preprocessedenvironment scene to contain the augmented object.

Object can stand for a primitive (polygon, triangle, solid, etc.), or acomplex object made out of primitives.

Hit point is a point where a ray intersects an object. Termed also HIP.

Visibility—given a set of obstacles in the Euclidean space, two pointsin the space are said to be visible to each other, if the line segmentthat joins them does not intersect any obstacles.

Scene, a collection of 3D models and lightsources in world space, intowhich a camera may be placed, describing a scene for 3D rendering. Scenemodel elements include geometric primitives: points or vertices; linesegments or edges; polygons or faces.

Clipping, in the context of computer graphics, is a method toselectively enable or disable rendering operations within a definedregion of interest.

The processes/devices and displays presented herein are not inherentlyrelated to any particular computer or other apparatus, unlessspecifically stated otherwise. Various general purpose systems may beused with programs in accordance with the teachings herein, or it mayprove convenient to construct a more specialized apparatus to performthe desired method. The desired structure for a variety of these systemswill appear in the description below. In addition, embodiments of thepresent invention are not described with reference to any particularprogramming language. It will be appreciated that a variety ofprogramming languages may be used to implement the teachings of theinventions as described herein.

Diverting from prior art technologies, aspects of the present inventionteach how to implement ray tracing at both a reduced computationalcomplexity and a high speed. One aspect of the invention relates to pathtracing, which is a high-quality ray tracing based on globalillumination. Its superior performance stems from a differenttechnological approach to solving the intersection between rays andscene objects. It is based on dynamically aligned structure (DAS), whichis a projection of parallel rays, used to carry secondary rays emittingfrom existing hit points. The DAS mechanism can be implemented either bya GPU (graphics processing unit) graphics pipeline, or by a CPU (centralprocessing unit). The mechanism can solve ray-triangle intersections byuse of the conventional graphics mechanism, replacing the expensivetraversals of accelerating structures in the prior art.

DAS mechanism. In one embodiment the DAS mechanism is applied to pathtracing, which is based on global illumination. Global illumination (orindirect illumination) takes into account not only the light that comesdirectly from a light source, but also light reflected by surfaces inthe scene, whether specular, diffuse, or semi-reflective. FIG. 3 depictsthe sampling of diffuse inter-reflection from the surroundingenvironment, at a given surface point. In order to achieve globalillumination on a diffuse surface, sampling rays must be shot from a hitpoint (HIP) 33. HIP is a result of a previous encounter between a ray(primary or secondary) and a triangle. The sampling is done by shootinga ray in a random direction within the boundaries of a hemisphere 31.The hemisphere is oriented such that its north pole is aligned with thesurface normal.

The basic mechanism of DAS is demonstrated in FIG. 4, where it is shownto be associated with a perspective projection; however, otherprojections such as parallel or inverse are conceivable as well. The DASstructure comprises a projection of rays passing through a cluster ofHIPs, e.g. 403, 405, 408, and targeting an object. The DAS rays that runinto HIPs are used as carriers for secondary rays. For example, a DASray that incidentally runs into a HIP 408 carries a secondary ray 406.This ray is falling within the HIP's hemisphere 407. A DAS ray that runsinto a HIP or a triangle consists of a starting point 400, and may havetwo intersection points, the first with a HIP 408 and the second with ascene object (triangle) 409. Additional secondary rays associated withthe same HIP 408 can be independently generated by additional DASstructures, carrying additional secondary rays in other directions,within the same hemisphere.

According to an embodiment of the present invention, the DAS projectioncan be implemented by a CPU software graphics pipeline, but the mostefficient device is the GPU hardware graphics pipeline. This well-knownart computer graphics pipeline is the process of turning a 3D model intowhat the computer displays. A graphics pipeline consists of twosubsystems: geometry and rasterization. First, all objects within theDAS frustum are transformed by the geometry sub-system, according to thecamera view. Then, in the raster sub-system, a ray/triangle intersectionpoint is selected by the Z-buffering mechanism. For example, the DAS ray402 in FIG. 4 is shot from the origin of the projection 400 andintersects with two objects; 408 and 409. Which of these two objects isselected, depends on the API directives (Direct3D or OpenGL) thatcontrol the Z-buffering.

An exemplary cluster of three existing HIPs, with their underlyingtriangles, are shown, 405, 408 and 403. Secondary rays of HIPs 405, 408are driven by the DAS structure. As an example, the carrier ray 402 runsinto HIP 408. From the point of encounter with the HIP and onwards, itbecomes a secondary ray 406, associated with the HIP, seeking forintersection 409. A DAS is relevant only to HIPs that have theirhemispheres oriented toward the projection, like 405 and 408, but not403. The DAS method is mathematically articulated as follows:

Let T be a tree-graph of d levels and let V be its vertices on top ofgeometries G in space.

Define V_(d)—vertices within V in level d.

Let C_(d) be a division of V_(d) to clusters.

We shall extend T to d+1 levels by finding V_(d+1):

Choose cluster c∈C_(d), with V_(d) _(c) vertices and define L_(c)—set ofmappings from V_(d) _(c) to V_(d+1) _(c) such that V_(d+1) _(c) areprojections of the vertices in V_(d) _(c) on top of G.

$V_{d + 1}:={\bigcup\limits_{c}V_{d + 1_{c}}}$

-   -   Note that L_(c) is a set of mappings from the same input, so        there can be several target vertices for any input vertex.

Instead of projecting every vertex v∈V_(d) _(c) on every possiblegeometry g E G in θ(|L_(c) |·|V|·≡G|), we project every possible g∈G onevery cluster c∈C_(d) in θ(|L_(c) |·(|V|+|G|)).

-   -   In R³ We can utilize traditional 3D graphics pipeline (raster        hardware) to achieve fast mappings (projections) in parallel.

We optimize C_(d)/L_(c) in throughput/overfitting to have:

-   -   Maximum number of vertices per cluster in average (throughput).    -   Minimum number of [discrete] projections of geometries fitting        all vertices (overfitting).    -   Preprocess/Runtime constraints.

L_(c) is chosen to have a pseudo-random output, representing a possiblesegment of distribution for each v∈V_(d) _(c) to simulate a physicalscenario.

Multiple DAS projections are shot at a scene or part thereof, inslightly different directions; each direction can be taken in a randommanner. As a result, multiple samples of the neighborhood can be takenat each HIP, for global illumination. This is illustrated in FIG. 5. HIP507 is a primary hit point created by the primary ray 508, shot from theimage pixel 500. The HIP 507 is visited by three subsequent DASprojections—501, 502 and 503. Each DAS carries one secondary ray for theHIP 507. Each of the three secondary rays delivers a different samplefrom the environment, i.e. from surfaces 504, 505, and 506,respectively.

There may be various ways to make use of the HIPs that are generated bysuccessive DAS projections. According to one embodiment, shown in FIG.6, all the newly generated HIPs contribute their data. In this example,four successive DAS projections are used. Assuming that 601 is a primaryHIP, generated previously by a primary shot from the image pixel 600,its first successor HIP 602 is a product of the first DAS projection.The second DAS projection, having a different direction, generates HIPchildren 603 and 604. The third DAS projection generates the HIPs 605,606, and 607. And then, one child HIP 608 is generated as a result ofthe fourth DAS projection. The light sampling from all HIPs must beaveraged and converged to the correct solution of rendering equation forthe image pixel 600, e.g. 608 and 605 converge into 603, which convergesinto 602, which in turn converges into 601. Finally, the primary HIP 601converges the aggregated sample values of all its children 602, 604, and607, and the result goes into pixel 600, as a partial contribution tothe pixel among other primary HIPs. A correct rendering equation for allconvergances should eventually generate a physically accurate image.

Secondary rays are meant to intersect with scene objects, asdemonstrated in FIGS. 7a-7c . FIG. 7a shows an initial scene consistingof two triangles 711, 712, and two primary HIPs 713, 714. In FIG. 7b theDAS projection 721 is shot towards the HIPs. The HIP 714 faces anopposite direction, therefore it is excluded from the current DAS. TheHIP 713, aligned positively with the projection, initiates a secondaryray. As shown further in FIG. 7c , the carrier ray associated with HIP713 is broken down into two parts; the early segment 731 and the mainsegment 732. The early segment 731 extends from the DAS origin up to theHIP 713. Its function is to pinpoint the HIP and its depth. Once thedepth Z_(HIP) is found, the main segment, from the HIP and forward tothe intersection point 733, plays the role of carrier for the secondaryray, searching for an intersection. It hits an object at 733, generatinga secondary HIP.

According to one embodiment of the present invention, the DAS projectionutilizes the Z-buffering mechanism of GPU, as illustrated in FIGS. 8aand 8b . The Z-buffering mechanism must discard objects all the waybefore the HIP, and starts seeking objects only from the HIP and on. Itis based on a selective use of the z-buffering mechanism of a GPU, e.g.the function glDepthMask of the graphics library of OpenGL. This is donein two separate rendering passes. In the first pass the HIPs arerendered as the only objects in the scene, disregarding the geometricdata and generating a HIP depth mask. In the second pass the HIP depthmask is used for rendering the scene geometry. The first pass is shownin FIG. 8a . The carrier ray 812, which overlays with the HIP 811, isbroken down into two segments, and is processed in two passes. The earlysegment, rendered during the first pass, extends from the camera 813 upto the HIP. The depth value of the HIP, Z_(HIP), is registered in theHIP depth mask 810. The depth value is kept for a later use forfiltering out all the objects on the way to the HIP, during the secondpass. In the second pass (FIG. 8b ), the geometric data is renderedstarting at the depth Z_(HIP), 811, e.g. the triangle 825 is ignored.The main segment, the carrier of the secondary ray, hits the triangle at823. The results of the second pass are stored in a render target 820.Rays that miss HIPs are entirely discarded, considered as early segmentsin their entirety. Once a render target is completed, the exactray/triangle intersection point 823 is found by inspecting the rendertarget at the u, v coordinates of the DAS carrier ray. The triangle ofintersection delivers essential data, such as color, light, normal,material, etc.

The DAS flowchart in FIG. 8c summarizes the method of generating andusing the DAS mechanism. The DAS projection targets an object (e.g. anaugmented object) or a sub-scene, passing through a cluster of HIPs, inorder to generate secondary rays for the HIPs. The DAS is shot twice.The first time it is shot at the HIP data only, ignoring the scenegeometric data, and producing a HIP depth mask 831. The second time anidentical DAS projection 832 is shot. This time the geometric data ofthe scene is rendered, ignoring the HIP data. The depth mask 810 isemployed to the starting point of secondary rays. Secondary rays aredriving on the DAS projection, seeking for intersection with thegeometric data. The rendering result, a render target, a 2D projectionof a 3D sub-scene, is basically a collection of all the intersectionpoints between the secondary rays and the geometric data of the scene.An intersection point, related directly to a specific HIP, can be foundby searching for coordinates u′,v′ on the render target, that match thecoordinates u,v of the HIP. The color and light values at theintersection points are fed back to the HIPs, delivering samples ofglobal illumination 833. Finally, the intersection points are stored inthe HIP repository as a next generation of HIPs 834.

Various cases of secondary rays are shown In FIG. 9, all carried by asingle DAS projection. Ray 900 consists of two segments. The earliersegment, extending from the camera 909 up to HIP 903, discards thetriangle 906, while the main segment encounters the triangle 907 at theintersection point 905. The secondary segment of ray 902 does not hitany object. Ray 901 fails to encounter a primary HIP, therefore it isconsidered as an early segment in its entirety, ignoring the triangle908.

The DAS mechanism of the present invention is implementable, among otherfields, in AR. One of its embodiments enables a localized path tracing,focused on rendering of one or more objects in the scene, and on aperfect integration between the augmented object(s) and the realenvironment. FIG. 10 shows an example of an augmented object, a Buddhastatue 101, standing on a real desk of a semi-reflective surface 107.What is needed to generate a photo-realistic appearance of the statue isnot just the image of the object 101, but its reflection 102 as well.The effect the augmented object may have on its real environment wouldresult in reflections, shadows and color bleeding that modify the presetenvironment. On the other hand, the impact of the environment on theaugmented object may result in lighting and reflection of the objectitself.

According to an embodiment of the present invention, the image of theobject and the object's reflection within the environment are generatedby two separate tasks, and the unified result is fed to the imagepixels.

Direct Imaging of an Augmented Object.

The basic image of the augmented object can be reconstructed just fromthe primary HIPs that cover the object's surface. However, for a globalillumination effect on the image, such as the environment reflected inthe object, secondary rays are required, shot from the object to itsenvironment. The rendering task of the augmented object 110 is shown inFIG. 11. For simplicity, it is described in a 2D drawing. The camera 113shoots primary rays 114 at the augmented object, seeking intersectionswith the object. These points of intersection become HIPs, meant to beused as a starting point for secondary rays, for global illumination.

The shootings of primary rays repeat, each with a slight change ofdirection, such that multi-sampling in image pixels is attained. Thechange of directions is done in a random fashion, in order to preventunwanted patterns in the image. Multi-sampling contributes to anantialised quality of the image. In FIG. 11 three primary shootings areshown 115, 116, and 117.

A truthful and integrated appearance of the augmented object in thescene is achieved by global illumination. The relevant environment forglobal illumination is sampled by secondary rays, shot from primary HIPstoward a relevant part of the scene. The relevant parts of the scene arethose visible from the camera by reflection in the object, if the objectis reflective. For example, such a relevant part can be the sub-scene123 in FIG. 12a , because its reflection in the object can be seen fromthe camera 127. The secondary rays are generated by DAS structures,either of perspective projection as in FIG. 12a , or parallel projectionas in FIG. 12b . In FIG. 12a the DAS projections are passing through theprimary HIPS (e.g. 128), targeting the sub-scene 123.

Since all the successive DAS projections are targeting the samesub-scene 123, this sub-scene can be clipped out from the full scene, toselectively enable rendering operations within a reduced region, thusminimizing the rendering process.

Each of the multiple DAS projections is generated randomly from aslightly different point of view and in a different direction, creatingmultiple secondary rays at each HIP. The use of randomness prevents theappearance of unwanted patterns in the image. Secondary rays sampleglobal illumination for a HIP (FIG. 3, 31), integrating between theobject and the environment. The sampled illumination impacts the imageaccording to the object's material and the level of its specularity ordiffuseness, e.g. if the object is reflective or partly reflective, itwill result in reflection of the environment in the object, or just anamount of background lighting if it is diffuse, creating the object'sresponse to the environment. The more projections of DAS, the bettercovering of global illumination. However, more projections may impairthe performance. Therefore, there is a trade-off between image qualityand performance.

The method of generating a direct image of an augmented object issummarized in a flowchart in FIG. 12c . First, multiple primaryprojections are shot from the camera (eye, view point) at the augmentedobject to generate a cluster of primary HIPs 1231. Then the parts of thescene to be targeted by secondary rays should be defined, possiblyclipped as a sub-scene 1232, and a reference point for DAS projectionsmust be set according to the chosen sub-scene 1236. Then, secondaryrays, generated by multiple DAS projections, are shot into the relevantsub-scene 1233. The result of the DAS projection is a render targettexture of the determined sub-scene. The seeking for an intersectionbetween a secondary ray and the determined sub-scene is done by matchingcoordinates of the related primary hit point and the render targettexture 1237. Each primary HIP is fed a light value of a correspondingintersection point between its secondary ray and the encounteredtriangle 1234. The above procedure can repeat if more than one sub-sceneare taken. Then, finally, the intersection points are added to the HIPrepository as a new generation of HIPs 1235. The processed samples ofcolor and light values, aggregated from all the primary hit points, areconverged into the image pixels, creating a full image of the augmentedobject affected by the three-dimensional scene.

Reflecting the Augmented Object.

Reflection of the object's image in the environment items isaccomplished by following a ray from the camera to the surface at thescene, and then bouncing toward the augmented object. Reflection on ashiny surface or tile enhances the photo-realistic effects of a 3Drendering. The extent of reflection depends on surface's reflectivity(the BRDF of the material). First, the reflective or semi-reflectivesurfaces (or items) in the real scene, which may reflect the augmentedobject, must be identified. Then we shoot primary rays at the surface inwhich the object is intended to be reflected, or part thereof,generating primary HIPs. From these HIPs, we shoot secondary raystargeting and sampling the augmented object. This way of generatingreflections is illustrated in FIGS. 13a and 13b . Primary HIPs coveringthe area of intended reflection are created by primary rays that areshot from the camera 133, through the image screen 130, toward the areaof reflection 134. The location and boundaries of the reflection area134 in surface 132 is determined according to the location of thecamera, the distance and the size of the augmented object 110, and theconsideration of the principal direction 131 according to the Snell law.The primary shots are repeated multiple times for multi-sampling in theimage pixels. Each successive time the primary projection is slightlydeviated randomly from the principal direction, such that each pixel ofthe image gets multiple samples. The surface of the reflection area 134becomes covered by a dense array of primary HIPs. The randomness of themulti-sampling prevents unwanted patterns in the resulting image.

FIG. 13b describes the generation of the reflected image by secondaryrays. The reflection of the augmented object 110 on the surface 132 isreconstructed from the sampled data in primary HIPs, which is gatheredby shooting secondary rays at the object. We use the geometric point136, which is the reflection of the camera 133 in the surface 132, as areference point for multiple DAS projections. Each projection is shotfrom a different point, randomly deviated from the reference point 136.The DAS 135, shown in FIG. 13b , originates in the reference point 136,directed along the axis 139, which is pointing towards the center of theaugmented object 110. The DAS carries secondary rays 138, all startingat the primary HIPs (e.g. 137) and targeting the augmented object.

In order to take a spectral sampling of the BRDF function at a HIP, themultiple DAS projections are randomly deviated from the reference DASprojection—the one that starts at the reference point and has an axis ofprojection directed at the center of the augmented object. Theinclination from the referenced DAS is done randomly, slightly deviatingfrom the reference point 142 and central axis 145, as shown in FIG. 14a. Three DAS projections are shown. Assuming that the reference DASoriginates exactly at the reference point 142, and its axis 145 takesthe central direction, then the two other DAS projections start at anearby points 141 and 143, and their axes 144 and 146 deviate from thecentral direction 145. As an example, we chose a HIP 140 from whichthree secondary rays are shot: 144, 145, and 146, each carried by adifferent DAS.

The relation between the deviation of a DAS secondary ray from thereference DAS, and its contribution to the aggregated light energy, isshown in FIG. 14b . It is strongly connected with the BRDF function 147of the surface material 132. Each of the three secondary rays 144, 145,and 146, are shot from the same HIP in a different direction, bounded bythe hemisphere of FIG. 3. As a result, its sampled data contributes tothe aggregated light energy, according to the BRDF function. Let'sassume that the secondary ray 145 goes in an exact Snell direction, thenit brings the maximal contribution at the peak of the BRDF function 147.Secondary rays 144 and 145 have a smaller contribution, depending on theBRDF value at the distance from the peak.

The way of generating a reflected image of an augmented object issummarized in the flowchart in FIG. 14c . First, the area in the realscene where the augmented object should reflect is determined 1431.Then, multiple primary projections are shot from the camera at the areaof reflection, generating a cluster of primary HIPs 1432. Next, thelocation of the reflected camera, as a point of reference for DASprojections, and the central axis directed toward the augmented object,must be calculated 1433. Then, secondary rays generated by DAS are shottoward the object. The multiple DAS projections are randomly tilted,deviating from the reference DAS 1434. Next, the light values sampled inthe intersection points are fed to their respective HIPs of origin 1435.Finally, the intersection points are added to the HIP repository as anew generation of HIPs 1436. These HIPs can serve for producing furthergenerations of secondary rays.

Color bleeding is the phenomenon in which objects or surfaces arecolored by reflection of indirect light from nearby surfaces. It is aglobal illumination algorithm in the sense that the illuminationarriving at a surface comes not just directly from the light sources,but also from other surfaces reflecting light. Color bleeding isviewpoint-independent, which makes it useful for all viewpoints. Thecolor bleeding effect in AR or VR would occur in the direct neighborhoodof the augmented object. An example of generating a color bleedingeffect by an embodiment of the present invention is illustrated in FIG.15a . The augmented object 154, standing on a substrate 152, is supposedto create a color bleeding effect on the real substrate 152. First, wedefine the boundary of a color bleeding patch around the center of theaugmented object where the color bleeding will appear. The size of thepatch depends on the involved materials, and the distance and the amountof light. Then we shoot primary rays from the camera 153 at the patch155, in the absence of the augmented object. A cluster of primary HIPsis created, covering the patch. The primary shots are repeated multipletimes, each time at a slight deviation from the principal direction 151.The principal direction 156 is from the camera toward the center of theobject's standing location.

FIG. 15b illustrates the use of secondary rays. The color bleedingeffect is reconstructed by sampling the object by secondary rays, whichare shot from the primary HIPs toward the object. The secondary rays aregenerated by DAS projections. The DAS projection, unlike in the cases ofreflection or of direct imaging of the augmented object, gets the shapeof an inverse projection 156. Multiple DAS projections are done, eachtime at a slight deviation from the principal direction. The samples ofthe object's surface are taken from the substrate, enabling acalculation of the amount of energy on the substrate, assuming that acorrect rendering equation is used.

The way of generating color bleeding is summarized in the flowchart ofFIG. 15c . First, the location and size of the color bleeding patch inthe scene is defined 1531. Then, multiple primary projections are shotfrom the camera at the color bleeding patch, to generate a cluster ofprimary HIPs 1532. Next, a reference point at the center of DASprojection is calculated 1533, as well as the required shape of theinverse projection 1534. Then secondary rays are shot by multiple DASprojections, each randomly deviated from the center of DAS projections1535, and the light values of intersection points are fed to primaryHIPs 1536. In color bleeding this is the only generation of HIPs.

Collecting Light Values.

The values of all samples in an HIP must be processed by a correctrendering equation for a physically accurate result. Parameters that aretaken into account are the surface material, the geometry of the scene,the active area of the hemisphere, and others. For a specific imagepixel, the light contribution of all HIPs generated by primary shotsfrom the pixel, and all their secondary successors, must be aggregated,processed, and converged into the source pixel for image. As shown inFIG. 16, the samplings from the object and from its environment areconverged into image pixels 164. The pixel receives inputs from aprimary HIP 165 on the surface of the augmented object, which in turncollects values from its successive generations. The pixel receives aswell input from a reflective HIP 161 and its successive generations. Theprocessed results of 165 and 161 are weighted, and then gathered in theimage pixel 164.

Implementation.

The core of the present invention is the DAS mechanism. When implementedin path tracing, it generates secondary rays and locates theirintersection with scene objects, excluding the use of acceleratingstructures of prior art. The DAS mechanism, which is based on aconventional raster graphics pipeline, is implementable either by a GPUhardware pipeline, or by a CPU software pipeline. The parallel structureof GPU makes it more efficient for graphics pipeline thangeneral-purpose CPU. A GPU is a specialized electronic circuit designedto accelerate the graphics pipeline. Where a CPU consists of a few coresfocused on sequential serial processing, GPUs pack thousands of smallercores designed for multitasking. There are two main types of graphicsprocessor: integrated and discrete. The DAS can be either utilized by aseparate component in a system (discrete GPU), or by an embedded GPU onthe CPU die (integrated GPU). Integrated GPU is used in embeddedsystems, mobile phones, personal computers, workstations, and gameconsoles.

The computing tasks that create the augmented objects and their visualcontext within the preset scene, as described in a great detailhereinbefore, are mostly based on graphics pipeline. For these tasks,the use of GPU is a great advantage. There is also additional task ofcollecting the sampled values of global illumination, processing thesevalues according to the rendering equation, and converging the resultsin the image pixels. The collecting task, being associated withconventional processing, can be implemented either by a CPU or by aGPGPU. There is also an additional task, associated with user's viewingdevice 171, shown in FIG. 17. For augmented reality those are wearablecomputer glasses that add information alongside or to what the wearersees. Typically this is achieved through an optical head-mounted display(OHMD) or embedded wireless glasses with transparent heads-up display(HUD) or AR overlay that has the capability of reflecting projecteddigital images as well as allowing the user to see through it. Forvirtual reality, the viewing device 171 can represent a virtual realityheadset that provides virtual reality for the wearer. VR headsets arewidely used with computer games but they are also used in otherapplications, including simulators and trainers. They comprise astereoscopic head-mounted display (providing separate images for eacheye), stereo sound, and head motion tracking sensors. Either way,component 171 must be interfaced to the computing platform by an APIsoftware, which is typically implemented by a CPU.

Consequently, the embodiments of the present invention call for acombined implementation of CPUs and GPUs, as generally shown in FIG. 17.The GPUs may stand for discrete graphics, integrated graphics or acombination of both: integrated graphics teaming with a discretegraphics.

Integrated graphics means that the GPU is integrated onto the CPU dieand shares memory with the processor. Since integrated GPUs rely on thesystem RAM, they don't have the computing power of their discretecounterparts that are contained on their own card and come equipped withtheir own memory, VRAM. Integrated GPU has a lower memory bandwidth fromsystem RAM, compared to discrete graphics cards between their VRAM andGPU core. This bandwidth is what is referred to as the memory bus andcan be performance limiting. Moreover, as a GPU is extremely memoryintensive, integrated processing may find itself competing with the CPUfor the relatively slow system RAM, as it has minimal or no dedicatedvideo memory. For the best possible graphics performance, discretegraphics chips outperform integrated GPUs.

On the other hand, sharing the same RAM memory can also be a benefit, asthe graphics cores on a multicore chip can better collaborate with CPUcores for exchange big data. The pure graphics tasks of imaging objects,reflections and color bleeding produce big data of light values, thatmust be collected and calculated for rendering equation by the CPUcores.

However, despite the performance advantage of discrete GPU, it isdesirable to use integrated GPU for implementation of the presentinvention in such applications as augmented reality, virtual reality,and computer games, for its better power-efficiency, affordability,portability and versatility. Integrated GPU, as a constituent of amulticore CPU chip, is used in embedded systems, mobile phones, tabletsand game consoles.

In addition to using discrete GPU or integrated GPU, it is also analternative of using a hybrid system having discrete and integratedGPUs, collaborating and alternating, depending on the task.

What is claimed is:
 1. A system for ray tracing an augmented object in athree-dimensional scene, utilizing a graphics pipeline, the systemcomprises: at least one graphics processor with memory, at least onegeneral purpose processors with memory, geometric data base of thethree-dimensional scene, render target memory; wherein in runtime, a.graphics processor shoots primary rendering projections at the augmentedobject, generating a cluster of primary hit points; and b. a sub-scenespace to be targeted by secondary rays is determined; and c. a referencepoint for secondary rendering projections is set; and d. a secondaryrendering projection is repeated multiple times, each time randomlytilted, when each time: i. secondary rays are generated utilizing thesecondary rendering projection, wherein the secondary renderingprojection is shot from vicinity of the reference point, through thecluster of the primary hit points, generating render target texture ofthe sub-scene; and ii. intersections of the secondary rays with thesub-scene are sought; and iii. the intersections are saved for nextgeneration of secondary rays; and iv. the light values are sampled atthe intersection points; and v. the sampled light values are fed back tothe primary hit points; e. the sampled light values are aggregated andprocessed for the primary hit points; and f. the processing results ofsampled light values are converged into image pixels.
 2. The system ofclaim 1, wherein the primary rendering projections are done by a meansof the graphics pipeline.
 3. The system of claim 1, wherein thesub-scene can be arbitrarily chosen as part of the three-dimensionalscene.
 4. The system of claim 1, wherein more than one sub-scenes can bedetermined.
 5. The system of claim 4, wherein each sub-scene must betargeted by separate sets of secondary rays.
 6. The system of claim 4,wherein for each sub-scene a different reference point is set.
 7. Thesystem of claim 1, wherein said sub-scene can be clipped out from thethree-dimensional scene.
 8. The system of claim 1, wherein the secondaryrendering projection is randomly tilted.
 9. The system of claim 1,wherein the secondary rendering projection can be of a perspectiveshape.
 10. The system of claim 1, wherein the secondary renderingprojection can be of a parallel shape.
 11. The system of claim 1,wherein the result of the secondary rendering projection is a rendertarget texture of the sub-scene.
 12. The system of claim 1, wherein theseeking for an intersection between a secondary ray and the determinedsub-scene is done by matching coordinates of the primary hit point andthe render target texture.
 13. The system of claim 1, wherein the savedintersection points of secondary rays can serve as a cluster of hitpoints for the next generation of secondary rays.
 14. The system ofclaim 1, wherein the processed samples of light values aggregated fromall the primary hit points, are converged into the image pixels, as partof a full image of the augmented object affected by thethree-dimensional scene.
 15. The system of claim 1, wherein the graphicsprocessor is a discrete GPU having a hardware graphics pipeline.
 16. Thesystem of claim 1, wherein the graphics processor is an integrated GPUhaving a hardware graphics pipeline.
 17. The system of claim 1, whereinthe graphics processor is a hybrid graphics system of at least oneintegrated GPU and at least one discrete GPU, all having a hardwaregraphics pipeline.
 18. The system of claim 1, wherein the generalpurpose processor is a multicore CPU having multiple processing cores.